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Abstract — This paper addresses the problem of coding a 
continuous random source correlated with another source which 
is only available at the decoder. The proposed approach is based 
on the extension of the channel coding concept of syndrome 
from the discrete into the continuous domain. If the correlation 
between the sources can be described by an additive Gaus- 
sian backward channel and capacity-achieving linear codes are 
employed, it is shown that the performance of the system is 
asymptotically close to the Wyner-Ziv bound. Even if such an 
additive channel is not Gaussian, the design procedure can fit the 
desired correlation and transmission rate. Experiments based on 
trellis-coded quantization show that the proposed system achieves 
a performance within 3-4 dB of the theoretical bound in the 0.5- 
3 bit/sample rate range for any Gaussian correlation, with a 
reasonable computational complexity. 

Index Terms — Distributed source coding, rate-distortion with 
side information, Wyner-Ziv coding, continuous channels, AWGN 
channel coding, syndrome-based coding, trellis-coded quantiza- 
tion. 



I. Introduction 

DISTRIBUTED SOURCE CODING addresses the prob- 
lem of coding the outcomes of multiple correlated 
sources independently, i.e. without allowing the respective 
encoders to communicate with each other Theoretical results 
on this topic appeared in the seventies, and showed that, if 
the joint distribution characterizing the correlation structure 
is known, there is no loss in performance with respect to 
(w.rt.) the case where the encoders can collaborate with each 
other This result goes under the name of the Slepian-Wolf 
coding theorem [1], [2]. A related problem is coding with 
side information at the decoder, in which the outcomes of 
a source have to be encoded and decoded within a given 
distortion under the condition that a second correlated source is 
only available at the decoder Theoretical performance bounds, 
given by Wyner and Ziv [3], show that in certain cases there 
is no loss in performance w.rt. the case where the encoder 
can access the side information as well. 

Coding methods that approach these theoretical limits are 
hence highly desirable, and have many practical applications. 
For example, they can be employed in sensor networks [4], 
where communication between the nodes not only requires 
an elaborate intersensor network, but also may be limited by 
bandwidth constraints. In addition to this natural distributed 
source coding application, these coding methods can be also 
used in video coding, not only for independent encoding of 
multiple correlated video sensors [5], but mostly to reduce the 
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encoding complexity w.rt. the classical, non-distributed, video 
coding solutions [6], [7]. Hyperspectral image compression is 
yet another field where distributed source coding principles 
can be taken into account [8], [9], [10]. 

Despite the limits of distributed source coding were the- 
oretically investigated more than thirty years ago, practical 
solutions to the problem have been proposed only in the 
last decade. All solutions stemmed from the connection that 
distributed source coding has to channel coding, as already 
pointed out by Wyner in its pioneering work [11]. Algorithms 
based on turbo and low-density parity-check (LDPC) codes 
[12], [13] have then appeared that approach the theoretical 
bounds in the discrete binary domain [14], [15]. 

To solve the coding with side information issue for continu- 
ous random sources, current proposals recast the problem into 
a binary domain or, at least, into a discrete domain, by using 
some kind of quantization. For example, in the distributed 
source coding using syndromes (DISCUS) system [16], [17], 
[18], the random variables are transformed into a discrete 
domain, in which cosets of some trellis codes are identified 
on top of the reconstruction codebook used by the quantizer, 
before actually coding them. The related idea of using nested 
codes for distributed source coding is discussed by Zamir et 
al. in [19], [20]. 

In this work it is proposed to solve the coding with side 
information problem for continuous random sources entirely 
in the continuous domain. By operating in this domain, the 
problem is recast into the traditional source coding problem 
of a continuous-valued syndrome that is closely related to the 
actual statistical correlation between the source and the side 
information. In a certain sense, the paper is an extension of 
the work in [19], since it is shown that, under some condition, 
it is not necessary for the codes to be nested for optimum 
performance, as first proved in [21]. 

The rest of the paper is organized as follows. In Section 
lU a review on linear block codes and lattices is given, and 
the concept of syndrome is introduced. Section |III] discusses 
the duality between the problems of distributed source coding 
and of channel coding. The discussion is presented from a 
perspective that enables us to directly utilize the concepts 
introduced in Section to solve the distributed source coding 
problem. This approach is presented in Section |IV] and the 
experimental results are discussed in Section [V] Conclusions 
are drawn in Section [VT] 

II. Linear Block Codes and Lattices 

In this section, the basic properties of linear block codes and 
lattices are reviewed. In particular, the focus is on the concept 
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of syndrome. After reading this section, it should be clear that 
the definition of such a concept catches some dualities between 
the two algebraic structures. 

A. Linear Block Codes and Syndromes 

Consider the Galois field F — GF{q), with q — for 
some prime number p > 2 and some m e {1,2,...}. An 
{n, kyiinear block code C over F is a fc-dimensional subspace 
of the vector space F" over F. Equivalently, 

C = I^J2a,g,:a,e¥, Vz = 1, 2, . . . , fcj , (1) 

for some set G = {gi}i=i,2,...,k of linearly independent vec- 
tors on F". It is worth noting that an [n! , k', m)-convolutional 
code over F, for a sufficiently large number L of consecutive 
n'-length output codewords, can be as well described by ([T]l, 
with n = n'L and k = k'L — m [22]. 

Being a subgroup of the additive group (F", +), C induces 
a partition of F" into cosets. In particular, the set of all cosets 
is the quotient group F"/C, and, since the cardinality of C 
is \C\ = q^, there are exactly |F"|/|C| = q'^/q^ = of 
them. To identify the coset to which an element of F" belongs, 
consider any linear application of F" with kernel equal to 
C into some other [n — fc) -dimensional vector space over F. 
Then, such a homomorphism of (F", +) automatically assigns 
a distinctive vector, called syndrome, to the elements belonging 
to the same coset. 

A simple way to construct such a linear application is as 
follows. Consider the orthogonal complement of C into F" 
(i.e. the dual code of C, with the canonical inner product on 
F"). Since is an [n — fc) -dimensional subspace of F", it 
is generated by some set H = {/ii}i=i,2....,n-fc of linearly 
independent vectors on F". Associate with each a £ F" 
the vector s//(a) on F"^''' whose canonical coordinates are 
the inner products {a, hi), i — l,2,...,n — fc. This vector 
represents the syndrome of a relative to the {n — k)x n parity 
check matrix H (whose rows contain the canonical coordinates 
of each hi). In addition, it is in principle straightforward to 
compute sii{o) since it can be simply obtained by matrix 
multiplication with H^. 

The syndrome plays an important role in minimum-distance 
decoding (using some distance function such as the Ham- 
ming distance) when it can be univocally associated with a 
minimum-weight element in each coset of F"/C. In this case 
the minimum-weight element that by subtraction leads to a 
codeword of C can be identified from the syndrome of any 
received codeword. Linear codes with good distance prop- 
erties permit to associate a unique minimum-weight element 
with most of the syndromes. When a syndrome without this 
property occurs, an error is detected but not corrected. 

B. Lattices and Continuous-Valued Syndromes 

Consider now the additive group of real numbers (M, +). 
An n-dimensional lattice A is a discrete subgroup of the 
Euclidean space K" that spans R" itself. The group (A, +) 



can be equivalently defined as 

A = 1^ fliWi : a, e Z, Vi = 1, 2, . . . , n| , 

for some set 2,...,™ of linearly independent vectors on 

Through the quotient group M"/A, any lattice induces a 
partition of R" into cosets. For practical reasons, it is useful to 
identify each coset using one of its elements. Hence, consider 
an injection / : M'VA R" such that 1{A) e A, VA G R'VA, 
and call labeling function of R"/A any such function. The 
fundamental region Ri (A) of A induced by I is then defined 
as the image of /, i.e. Ri[K) = ?(M"/A) C R". Clearly, the set 
of translates {i?i(A)+& : & G A} forms a regular tessellation of 
R", and hence the volume of i?;(A) equals V{K), the volume 
of n-space per point of A. 

Since M" is a normed space under the usual i^-norm, it 
is common to take elements with minimum norm as coset 
representatives. A labeling function V such that 

\V{A)\ < \a\, VAeR"/A,aG A 

defines then a fundamental region Rv{h) known as funda- 
mental Voronoi region. The corresponding tessellation of M" 
consists of decision regions for a minimum-distance quantizer 
(or decoder) that uses A as codebook. 

The role of the labeling function becomes evident when 
the induced group structure on Ri{A) is considered. The 
labeling function is in fact invertible in Ri{A), and hence, 
upon defining the sum operation in Ri (A) as 

a + p = l{l-\a) + r^{(])) , Va,/3ei?KA) , (2) 

I is an isomorphism. Denote with i/ the natural homomorphism 
ly : R" -> R"/A, and define the function si = I o ly : R" 
Ri{A). Once a labeling function I is defined, this homomor- 
phism identifies indeed the coset to which any element of M" 
belongs. As a consequence, s;(a) represents the continuous- 
valued syndrome of a £ M", by analogy with the role of the 
traditional syndrome in linear codes. 

The continuous-valued syndrome satisfies the following 
properties: 

si{a + X) = s;(a), Va e R",A e A ; (3) 
si{a) = a, Vaei?;(A); (4) 
si{si{a)) = s,(a), VaeR"; (5) 
siia + b) = .Si(a) + s,(&), Va,6eR" . (6) 

In particular, ^ and ^ follow directly from the definition 
of si, and state that si is periodic and that the restriction 
si\R,{A) '■ Ri{A) Ri{A) of si is an identity respectively; 
(|5]i follows directly from dU, and states that si is idempotent, 
i.e. that s™ ~ si, Vm ~ 1,2,...; (|6]l, in which the sum 
on the right-hand side is intended as defined in (|2]i, follows 
from the fact that si is a homomorphism. As a remark, 
si : R" (M",+) is not a homomorphisn|l| since i?;(A) 

'For the sake of clarity, the continuous-valued syndrome s;( ), when 
intended as belonging to (R", +) rather than (Ri (A), +), will be hereinafter 
indicated as Si(-). 
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is not a subgroup of (R", +)0 Hence, in general, if both the 
sums are taken in (R", +), si{a + b) ^ si{a) + si{b). However, 
it is straightforward to show that 



Si{a + b) = Si (S/(a) + S/(6)) , Va, & G 



(7) 



which in turn shows that si (a + b) ~ si (a) + si{b) if and only 
if siia) + Si{b) e RiiA). 

In order to evaluate the continuous-valued syndrome for a 
given a G M", it is immediate to verify that the syndrome 
sv{a) £ i?y(A) relative to a fundamental Voronoi region 
can be obtained as quantization error of a minimum-distance 
quantizer that uses A as codebook. In particular, defining 

QA(a) = A e A : |A - a| < |7 - a|, V7 G A 

as the closest lattice point to a (with the further condition 
(a — A) G Rv (A) in case of ambiguity), we have 



sv{a) 



(8) 



In this case, as the traditional syndrome, the continuous-valued 
syndrome of a identifies the minimum-norm element that, 
subtracted to a, leads to an element (the closest) of A. 

III. Distributed Source Coding and Channel 
Coding 

After a brief review of the well known concepts of dis- 
tributed source coding and coding with side information, this 
section discusses how these problems are intertwined with the 
more traditional channel coding problem. In particular, this 
fact is due to the existence of a virtual correlation channel, 
which can be seen both as forward or backward channel. 

A. Distributed Source Coding and Coding with Side Informa- 
tion 

The notion of distributed source coding refers to the 
problem of coding the outcomes of a random vector X = 

fx 



^\i=1.2,....N 



by using N independent encoders that cannot 
collaborate with each other, but whose respective N outputs 
are jointly fed to a single decoder. Assume that the random 
variables Xi have a discrete alphabet. Then, by the source 
coding theorem [1], perfect reconstruction is asymptotically 
achieved if each encoder can communicate with a rate greater 
or equal than the entropy rate Hoo{Xi) of Xi. However, be- 
cause of joint decoding, this is no longer a necessary condition. 
Slepian and Wolf showed that in case of independent and 
identically distributed outcomes with N — 2 (see Fig. [U, the 
necessary (and sufficient) conditions are [2] 

Ri > H{Xi\X2), 
R2 > H{X2\Xi), 
R1+R2 > H{Xi,X2), 

where H{-\-) and H{-, •) denote the conditional and the joint 
entropy respectively. 

^For the same reason, while the identity function is obviously a 

homomoi'phism -R;(A) — > (i?;(A), +) (in particular, an automorphism), it is 
not a homomorphism Ri{A) — * (M",+). 
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Eig. 1. Distributed coding of two random sources. 
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Fig. 2. Coding with side information at the decoder 



In such a case, denote Xi, X2, and Ri with X, Y, and R 
respectively. If R2 > H{Y), i.e. if the decoder can perfectly 
reconstruct the outcomes of Y upon receiving the output of 
the second encoder (see Fig. |2|i, the first encoder just needs 
to communicate with a rate equal to H{X\Y) < H{X) for 
perfect reconstruction, since the shaded decoder in Fig. |2] can 
rely on the side information Y correlated with X. Given a 
certain upper bound D to the distortion between X and X 
(that can be discrete or continuous random variables), the 
general problem of finding the minimum achievable value 
for R goes under the name of coding with side information 
at the decoder. This problem was solved by Wyner and Ziv 
[3], and it turned out that the obtained rate-distortion function 
(13) is in general greater or equal than the rate-distortion 
function Rx\y{D) corresponding to the case where Y is as 
well accessible by the encoder. 

B. Connections to Channel Coding 

Since the ultimate performance limits shown in [2], [3] 
were obtained with non-constructive proofs, effective, practical 
solutions have not appeared until recently. Current approaches, 
that are getting closer and closer to the theoretical limits [4], 
have stemmed from the connection that distributed source 
coding has to channel coding [11]. 

In fact, the statistical dependence between X and Y can 
be exactly characterized in terms of a virtual correlation 
channel. This channel can be defined in two equivalent ways, 
as follows. 

1) Forward channel (FCH): the complete statistical descrip- 
tion of the tuple (X, Y) is given by the probability mass 
functiorll ipmf) p{x) and by the conditional pmf p{y\x). 
Hence, Y can be seen as the output of the stochastic 
channel described by p{y\x) into which X, distributed 
according to p{x), is fed as input. 

2) Backward channel (BCH): alternatively, the complete 
statistical description of {X, Y) can be given by the 

'if the random variables are not discrete, the probability mass functions 
p( ) can be replaced with probabihty density functions ipdf) /(•). 
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Fig. 3. Forward additive cliannel (Nf is independent of X). Fig. 4. Backward additive channel (TV is independent of Y) equivalent to 

the forward channel of Fig. [5] 



pmf p{y) and by the conditional pmf p{x\y). In this 
characterization, X is seen as the output of the stochastic 
channel described by p{x\y) whose input is Y, dis- 
tributed according to p{y). 

In literature the FCH interpretation is commonly given. In 
the coding with side information problem, according to this 
interpretation, a noisy observation Y of the unknown X is 
given to the decoder together with some prior information 
about X, provided by the encoder. Being aware of the FCH 
statistics, the encoder must hence provide to the decoder the 
smallest number of independent constraints which allow for 
reconstruction within the desired distortion. 

In practice, in the discrete case the alphabet of both X 
and Y coincide with a Galois field F, and the FCH can be 
described as additive channel. This implies that Y = X + 
Nf, with Nf a random variable (noise) independent of X and 
distributed according to some pmf p{nf) (see Fig. [3]0. The 
connection to channel coding then comes from the fact that 
there may exist a zero-error (n, fc)-linear code which achieves 
the capacity C = log2 |F| — H{Nf) (bits per channel use) of 
the FCH, i.e. such that fc/rt-logj |F| = C. In this case X can be 
perfectly reconstructed once the coset to which each successive 
n-tuple of outcomes of X belongs is signalled to the decoder. 
This is simply accomplished by decoding Y into the signalled 
coset. If X is uniformly distributed on F this communication 
requires a rate R = {n — k)/n ■ log2 |F| (bits per channel use), 
i.e. it achieves the Slepian-Wolf Hmit H{X\Y) = H{Y\X) = 
H{Nf) = log2 |F| - C. 

As discussed in Section III-AI the syndrome sh {x) can be 
used to inform the decoder about the coset to which the un- 
known belongs. Details and experimental results regarding this 
distributed coding strategy called distributed source coding 
using syndromes (DISCUS) can be found in [4], [16], [17], 
[18]. 

Once the decoder has the side information, the remaining 
information needed for perfect reconstruction should actually 
regard the noise only. The fact that in the previous approach 
the communication rate equals exactly R = H{Nf) seems to 
confirm this observation. However, within the additive FCH 
interpretation there is no apparent relation between the noise 
and the output of the encoder, which is indeed driven by a 

^It is worth to remark that this characterization defines a symmetric channel 
in the sense given in [1]. Consequently, H{Y) > H(X), where, unless Nf is 
pseudo-aleatory, the equal sign holds if and only if X is uniformly distributed 
(this condition is necessary in order to achieve the capacity of any symmetric 
channel and implies that Y is uniformly distributed as well). The additive 
channel, however, captures neither all possible forms of correlation nor even 
all possible symmetric coirelation channels. 



signal perfectly independent of the noise itself. 

According to the BCH interpretation, instead, the input of 
the encoder can be seen as a direct observation of the unknown 
noise introduced by the channel, but measured with some offset 
(that is input into the BCH) known at the decoder only. The 
encoder, hence, given the BCH statistics, should provide the 
decoder with the smallest amount of information that allows 
for reconstruction of the unknown (i.e. of the noise) within 
the desired distortion. Obviously, it should operate in a way 
such that any allowable offset does not increase the amount of 
information needed. 

Again, in the discrete case the BCH can be usually described 
as additive channel over the finite field F. This implies that 
X ^ Y + N (the term offset used for the side information 
becomes now clearer), with N a noise independent of Y 
and distributed according to some pmf p{n) (see Fig. |4]i. If 
there exist a zero-error {n, fc)-linear code which achieves the 
capacity C ~ log2 |F| — H{N) (bits per channel use) of the 
BCH (i.e. such that k/n ■ logj |F| = C), X can be perfecdy 
reconstructed once the coset to which each successive rt-tuple 
of outcomes of X belongs is signalled to the decoder. By 
knowing the offset, in fact, the decoder easily computes first 
the coset to which the noise belongs, and then the noise 
itself. Since this communication requires a rate up to i? = 
(n — k)/n ■ log2 |F| (bits per channel use), the Slepian-Wolf 
Hmit H{X\Y) = H{N) = log2 |F| - (7 is exactly achieved. 

Apparently, in the discrete case examined above, if both the 
additive FCH and BCH interpretations are applicable it is not 
yet clear why one should prefer one over the other In this 
paper it is claimed that the BCH interpretation is preferable 
because of the following. 

• The existence of a good linear code for the additive FCH 
does not guarantee the achievability of the Slepian-Wolf 
limit when X is not uniformly distributed. For example, 
it is always possible to assign non-uniform probabilities 
p{x) in a way such that each coset is still equiprobable. 
Consequently, the transmission rate needed will still equal 
H{Nf), even if in this case H{Y) > H{X) implies 
hIx\Y) = X{Y\X) + H{X) - H{Y) < H{Y\X) ^ 
H{Nf). 

• If a good linear code for the additive BCH exists, then the 
Slepian-Wolf limit H{X\Y) = H{N) is always achieved, 
independently of the pmf of Y. Moreover, during the 
design of the distributed coding system, it is not necessary 
to know p{y), but only p{n), i.e. the BCH statistics. 

« Finding a good linear code for the additive BCH is 
equivalent to looking for a good linear encoder for N 
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(i.e. a good linear encoder for X when Y = 0). In fact, 
the latter search implies to find a sufficiently large n such 
that n-tuples of N distribute uniformly into the typical 
set ^("' C F", and a linear function s : F" ^ F"-'^ 
whose restriction to ^'"^ is one-to-one into F"^*' (and 
hence with k such that |F|"-''' = ~ 2"-f^(^)). The 

dual code of s(F") is then a good linear code for the 
additive BCH. 

Within the BCH interpretation, the encoder design is hence 
directly tailored to the noise N . Then, since encoding is a 
linear operation, any ojfset known by the decoder will not 
harm the performance of the system. In fact, the decoder can 
easily remove its effect on the information received by the 
encoder, which is still uniformly distributed on F"^''', and 
hence requires the same transmission rate. 

It is worth to point out that in the discrete and finite 
case both forward and backward interpretations give additive 
channels (with input-noise independence) if and only if X 
and Y are uniformly distributed, unless Nf (or N) is pseudo- 
aleatory (in which case the channel is deterministic and one- 
to-one). In particular, if an additive FCH exists and X is 
uniformly distributed, then N ~ —Nf is independent of Y, 
which in turn is uniformly distributed. Viceversa, if an additive 
BCH exists and Y is uniformly distributed, then Nf = —N 
is independent of X, which in turn is uniformly distributed. 
In practice, if iV = —Nf, the two schemes in Fig. [3] and 
Fig. |4] are equivalent. The point here is that any good linear 
code for the additive FCH turns out to be equally good for 
the BCH. In other words, the DISCUS system that is tailored 
for an additive FCH with a uniformly distributed source is 
also optimal for the corresponding BCH with any desired 
side information distribution. Even if in the discrete case it 
is not necessary to explicitly design codes using the BCH 
interpretation, this approach will turn out to be more useful 
for the continuous channels discussed in the following. 

C. Continuous Correlation Channels 

Assuming that X and Y are continuous random variables 
that take values on the Euclidean space R, the virtual channel 
(FCH or BCH) that describes the mutual correlation is a 
continuous channel. In this case, in general, as _D ^ 0+, 
> Rx\y{D) +00, i.e. in the problem of coding 
with side information perfect reconstruction is practically not 
achievable, unless we allow the encoder to communicate at an 
infinite rate. In the following, assuming for a while that this 
is possible, the generalization to this domain of the results of 
the previous sub-section is discussecH 

In particular, assume that the FCH is actually an additive 
channel such that Yf = X + Nf, with Nf independent of 

^This generalization is not straightforward. The differences do not essen- 
tially follow from the fact that the domain is continuous, but rather arise 
because the space K has a non-finite measure. In this case, the uniform 
distribution cannot be defined on the entire alphabet, and hence the capacity 
of the additive channels cannot be achieved by such a pdf. Consequently, the 
pdf of the input and of the output of the channel when the capacity is achieved 
differ. Instead, there exist additive channels in a continuous but ^niYe-measure 
domain (as for example the mod-A channel defined in [23]) where the duality 
with the discrete and finite case would be more strict. 



Fig. 5. Forward continuous additive channel (Nf is uncorrelated with X). 



X (see Fig. |5]q Suppose that a lattice A exists in M" whose 
intersection with a bounded set A C M" defines a capacity- 
achieving code Ah (subject to some cost constraint) for the 
FCH. The volume of n-space per point of A approximates 
the volume of the typical set of Nf, because in each coset 
of M"/A there is a unique noise representative. Perfect re- 
construction (with probability close to one) is achieved by 
channel-decoding Yf into the coset signalled to the decoder 
as the coset to which any n-tuple of outcomes of X belongs. 
It would be for example possible to reconstruct X with 
negligible probability of error by transmitting a continuous- 
valued syndrome si{x) as defined in Section Hl-BI 

However, differently from the discrete case, Yf and Nf are 
neither independent nor uncoiTelated {E\YfNf] = E[N'j] ^ 
0), not even when the pdf of X achieves channel capacity. 
Hence, the information brought by Yf about the noise may 
relax the need for transmission of as many different messages 
as the possible realizations of Nf. In the example of an 
additive Gaussian channel with N f G A/'(0, crp, the volume of 
71-space per point of A is approximately ^(A) ~ 2"''*^^^'-' = 
2n/iG(ff/)^ where h{-) and /ig(-) denote the differential entropy 
and the differential entropy of the Gaussian random variable 
(i.e. the upper bound of the differential entropy under the spec- 
ified power constraint) respectively. Invoking a sphere-packing 
argument, if A" G A/'(0, fi^) (which achieves the capacity under 
the constraint i?[A"^] < cr^) Yf is used at the decoder to 
discriminate, on average, between about 2"'''=('"- V^(A) ^ 
2t i°g2('T^/f^/) different n-tuples. But actually the number of 
distinct messages per n channel uses that could be reliably 
transmitted through the FCH under the same power constraint 
is 2"*-^ = 2^ '°82(i+'^x/'^/) > 2^ '°^2(f^x/'^/)^ 

In other words, there must exist a denser code A|, (which 
is assumed to be a bounded subset of a lattice A' C K") 
such that the relative coset information is still sufficient 
for perfect reconstruction. Since at least a coset of M"/A' 
contains more than one representative of Nf, for this code 
the algorithm sketched above (channel-decoding of Yf into 
the coset signalled by the encoder) no longer leads to perfect 
reconstruction. 

As noted in the previous section, a slightly different perspec- 
tive is possible when the correlation is described in terms of 
a BCH. In the continuous domain, under the condition that X 
and Nf are uncorrelated, any additive FCH can be interpreted 

*It is hereinafter assumed that the channel is not deterministic and that, 
without loss of generality, the continuous random variables have zero mean. 
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Fig. 6. Backward continuous additive channel (TV is uncorrelated with Y) 
equivalent to the forward channel of Fig. [s] 



as additive BCH as follows. Define 



with 



Y^aYf 



E[X^ 



(9) 



<1, 



E[X^] + E[Nj] 
as the optimum linear predictor of X from Yf. Consequently, 

N ^ X -Y = X - aYf = {1~ a)X - aNj 

is uncorrelated with Y, and X = Y + N. Hence, if Yf is 
multiplied by a, then the forward channel of Fig. |5] can be 
interpreted as a backward channel (see Fig. |6ll, with input, 
noise and output variances equal to 



E[Y^] 
E[N^] 
E[X^] 



= a^E[Yj] = aE[X^] 
= E[X^] - E\Y'^] = (1 - 



(10) 

x)E[X^]=aE[Nl}\) 



a{E[X^] + E[Nj]) = aE[Yf] 



Moreover, if X and Nf are independent and X is distributed 
according to the pdf that achieves the FCH capacity, then it 
is reasonable for N to be independent of Y . This is certainly 
true for additive Gaussian channels. 

Using this interpretation, and assuming that the pdf of X 
is the one that achieves the FCH capacity, a code K'^ is found 
such that signalling the coset of ]R"/A' to which any n-tuple of 
outcomes of X belongs is sufficient for perfect reconstruction 
at the decoder. In the Gaussian case with Nj e A/'(0, cr?). 



for example, the scaled code A|, 



t^/'^A/, can be used. 



Observe that it represents a capacity-achieving code for the 
BCH with the power constraint < acr^ (that Y as 

defined in (|9]l satisfies), which has the same capacity C = 
ilog2(l + a'^/aj) of the FCH with the power constraint 
E[X'^] < a^. Hence, with probability close to one, any other 
n-dimensional realization of the noise N belongs to a different 
coset of R"/A', such that in each coset there is a unique 
noise representative. Being independent of N, Y (Yf) does 
not give any other useful information about A^. As in the 
discrete case, since the information about the noise can be 
linearly formed at the encoder with an offset known at the 
decoder, perfect reconstruction is achieved. In particular, this 
is accomplished by channel-decoding Y (in place of Yf) at 
the decoder. Now that the volume of n-space ^er point of A' 
is approximately V{K') - 2"''(^' = 2"^°("'^/\ it turns out 
that the number of distinct messages per n channel uses that 



could be reliably transmitted through the correlation channel 
is correctly 2"''g('^')/V"(A') ~ 2? = 2"^. 

While in the discrete and finite case any good linear code 
for the additive FCH turns out to be equally good for the 
BCH, in the continuous case the coset information sent to 
the decoder should be relative to a code, tailored to the BCH 
statistics, which is denser than the former This approach will 
be discussed in the next section, where it will be showed 
that if a good channel code is available for the additive 
Gaussian BCH, then the rate-distortion function can 
be approximately achieved by sending a quantized version of 
the continuous-valued syndrome, independently of the pdf of 
Y. 

IV. Continuous-Valued Syndrome-Based 
Distributed Source Coding 

In the last fifteen years, the binary codes that have been in- 
troduced in the literature, such as the turbo-codes [12] and the 
low-density parity-check (LDPC) codes [13], come very close 
to the channel capacity of the binary additive channel with a 
reasonable computational complexity. But, in many distributed 
source coding applications, X and Y take binary values, and 
their correlation can be analogously described by a virtual 
additive channel. Hence, since soon after the discovering of 
the strong connection to channel coding discussed in Section 
IIII-BI several approaches have been successfully proposed for 
the problem of distributed source coding based on these linear 
channel codes [14], [15]. 

Following the success of these coding algorithms in the 
binary domain, if continuous (or discrete, but non-binary) 
random variables have to be coded, a transformation into the 
binary domain is usually applied prior to the actual coding 
operation, for example by first quantizing the variables and 
by then scanning their values by bit-planes [6]. 

Otherwise, it is possible to consider non-binary discrete 
linear codes, such as the trellis codes ([24], [25]), to directly 
code discrete random variables or quantized versions of con- 
tinuous random variables. For example, the DISCUS system 
[17] is actually concerned with the problem of coding with side 
information a continuous variable X, which is first quantized 
into a discrete domain, in which cosets of some trellis code 
[26] are identified. For any n-dimensional realization x of X, 
the encoder sends the syndrome sh{w) that identifies the coset 
to which the quantized version w of x belongs (see Fig. |7]l. 
Since the version w of w reconstructed at the decoder lies 
in the discrete domain, then a minimum square-error (MSE) 
estimate is found, using as well the same side information 
y which by channel-decoding led to w. 

While these coding systems achieve somewhat a good 
performance, the interplay between the quantizer and the 
syndrome former [17], [27] does not usually lead to a straight- 
forward optimal design procedure once the correlation and 
the available transmission rate are known. According to the 
discussion in Section ITlI-CI by using a continuous-valued syn- 
drome it is instead possible to form a syndrome which directly 
depends on the continuous realization to be coded. If the 
syndrome formation is exactly tailored to the BCH correlation. 
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Fig. 7. Coding with side information at the decoder, according to the DISCUS 
system. 
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Fig. 8. Coding with side information at the decoder using continuous-valued 
syndromes. 



then upon receiving that information perfect reconstruction is 
possible. Otherwise, if that information cannot be entirely sent 
due to some transmission rate constrain, it will be still possible 
to send a quantized version of it. The difference here is that the 
design of the system exactly depends on the knowledge of the 
correlation (responsible for the syndrome formation) and on 
the knowledge of the available transmission rate (responsible 
for the syndrome quantization). 

The proposed coding system is sketched in Fig. [8] The 
continuous-valued syndrome sv{x) is formed in correspon- 
dence of any n-dimensional realization x of X, and an ap- 
proximation sv{x) is received at the decoder, which estimates 
X by channel-decoding of y (that represents the input of the 
BCH) and finally reconstructs Xr as a MSE estimate of x from 
X and y. 

In the case that a capacity achieving linear code (i.e. a 
bounded subset of a lattice A) exists for the BCH and 
that the corresponding continuous-valued syndrome sv{x) is 
correctly received at the decoder, perfect reconstruction of x 
is asymptotically achieved when n ^ oo. This is shown by 
the following expressions 



y + n 

y + Sy {n) 

y + sv(x-y) 

y + sv {sv{x) + sv{-y)) 



(12) 
(13) 



where (fT2b follows from (01 and from the fact that all noise 
realizations tend to lie uniformly in the fundamental Voronoi 
region of A, and ( fT3] l follows from the property of the sum (|7]). 
Even if equation ( fT3] l already shows the sufficiency of Sv{x) 
for perfect reconstruction, from an operative point of view it 



is useful to note that 

X = y + sv {sv{x) - y - Qhi-y)) (14) 
= y + sv {sv{x) ~ y) (15) 
= sv{x) - Qk{sv{x) - y) , (16) 

where (fT4l i follows from the operative definition of the 
continuous-valued syndrome ([8]), ( fTSl l follows from the pe- 
riodicity (O of sy( ), and ( fTSI ), again, from the definition 
(O. Then, a single quantization operation is sufficient at the 
decoder for perfect reconstruction. 

Assuming now that the channel-decoding algorithm is the 
same even if the received information from the encoder 
consist of a quantized (noisy) version of the continuous-valued 
syndrome, at the decoder the following hold 

X = y + Sv (svix) + q + sv{-y)) (17) 

= y + sv {sv(x) + sv{q) + svi-y)) (18) 

= y + Sv{x + q-y) (19) 

= y + sv{n + q) 

= x+{q-QA{n + q)) = x + {q + qoi) , (20) 

where: (fTTb is obtained from (fTSl l by substitution of sv{x) 
with sv{x) — sy{x) + q (q is m n-dimensional reaUzation 
of a random variable Q); dTsl l follows from Q and from the 
reasonable assumption that all the realizations of q lie in same 
region as n; ( fT9l ) and ( l20t follow, again, from the sum-property 
(|7]i and from the definition ([8]) respectively. 

The total reconstruction error qt ~ q + qoi is hence the 
sum of the granular error q, and of the overload error qoi = 
—Qh{n + <z)- If <z is negligible w.r.t. n, and it is reasonable 
that this happens at high transmission rates (recall that if 
Ab is capacity-achieving sv{X) is distributed as N), then 
the probability of qoi being not zero (that is called error 
probability Pe) is negligible and qt ~ q. Hence, the same 
additive error Q would impair both the syndrome and the 
source at the decoder (as it was shown in a different way in 
[21]). This in turn shows that the operational rate-distortion 
function obtained by the proposed coding system i?^|y(_D) 
satisfies at high rates 

where Rs^ [D] is the operational rate-distortion function 
relative to the source encoder used to code the continuous- 
valued syndrome. Again, since sv{X) is distributed as N, 
the ultimate performance achievable is then represented by 
Rn{D), the rate-distortion function relative to the noise added 
by the BCH. 

This is particularly interesting when the noise N (which is 
the difference of X and Y) is Gaussian (and of course inde- 
pendent of Y), i.e. N G J\f{0, u^). In this case, as shown first 
in [28] and then in [29], R*j^^y(^) ^ Rn{D) independently 
of the distribution of Y (in particular, without requiring Y to 
be Gaussian as first assumed in [3]). Consequently, 

R*x\YiD)>RNiD)^R*^^YiD) , 

'An error occurs each time some coordinate of the n-dimensional vector 
Qa(" + q) is not 0. 
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i.e. in the Gaussian case and at high rates, the ultimate perfor- 
mance of the proposed coding algorithm is the rate-distortion 
function with side information at the decoder itself. This resuh 
is achieved upon utilizing a capacity-achieving channel code 
Af, (i.e. upon utilizing a rate-distortion optimal source code 
[28]) for syndrome formation, and a rate-distortion optimal 
source coder for syndrome encoding. 

In case the rate is not sufficiently high for this to hold, 
Pe 7^ and an MSE estimate can reduce the global error. In 
particular, observing that X — Y + N (with N independent of 
Y) and X = X -\- Qt (with Qt assumed independent of both 
Y and N and with variance ), it is perfectly correct to use 



Xr — 



-X 



-Y 



(21) 



processed is n = 1000, and the tested codes are the ones in 
[26] relative to 8-, 64-, and 256-state trellises. 

The experimental second moment per dimension of the 
Voronoi region relative to the TCQ codes measured in [26] 
is used to determine the scaling factor a in a way such 
that the continuous-valued syndrome variance equals the noise 
variance cr^ multiplied by a volumetric factor K. This factor 
takes into account for the non-optimality of the TCQ codes for 
channel coding and it is numerically estimated. For a capacity- 
achieving code, sv{X) would be distributed as N and hence 
have the same variance {K = 1); in practice, using sub-optimal 
codes, the normalized volume (per two dimensions) of the non- 
spherical Voronoi region must be larger (K > 1) for optimal 
performance. 



as optimal Unear prediction of X from X and Y at the 
decoder; the residual error variance cr,^ would satisfy then 
l/a^ = 1/al + 1/al [30]. 

The existence of effective source coders for Gaussian 
random variables, such as the trellis-coded quantizers [26], 
suggests the feasibility of their application in this framework, 
in the syndrome encoder as well as in the channel decoder 
This application, with the relative experiments, is discussed in 
the next section. 



V. Experiments 

In this section, the problem of coding a source when another 
one is available only at the decoder is targeted with the 
approach outlined in Section |IV] In particular, it is assumed 
that the correlation is modeled by a backward Gaussian 
additive channel (Fig. |6]l with N G J\f{0,a^). The variance 
(j^ of the Gaussian noise to be generated is chosen in order to 
guarantee a given backward correlation signal-to-noise ratio 
(BC-SNR), i.e. a certain ratio between the variance ay of the 
side information Y and crf^. In the experiments, is fixed (and 
equals 1), but both the Gaussian and the uniform distribution 
has been tested for Y, obtaining as expected exactly the same 
results. 

As a remark, note that if F e Af{0, ay), then X £ M{0, al), 
with cr^ — "'y + '''n' '^he simulated additive BCH can 
be assumed as derived from a Gaussian additive FCH with 
Gaussian input (as discussed in Section IIII-Cl l. which is the 
problem treated in [17]. For comparison purposes, observe 
that the correlation signal-to-noise ratio (C-SNR) of the 
corresponding FCH equals the BC-SNR of the simulated BCH 
since al/a'j = ay/a^^, as given by ([TOll and (fTTlFi 



A. Continuous-Valued Syndrome Formation 

As trellis-coded quantization (TCQ) based on the partition 
aZ/4aZ [26] defines a geometrically uniform source code 
[31], and in particular a lattice A [24], the encoder computes 
the continuous-valued syndrome of the source as quantization 
error after TCQ, as given in The number of samples jointly 

*The only difference between the two approaches is then in the fact that 
(T^ is kept constant in [17], while it is ct^ that is kept constant in this paper. 



B. Continuous-Valued Syndrome Coding 

Any suitable source coding algorithm can be used to code 
the n-dimensional continuous-valued syndrome sv{x) into its 
approximation sv{x) at the desired transmission rate. In this 
paper, since the focus is on practical applications, the target 
rate R is chosen in the range 0.5 3.0 bit/sample. Again, 
TCQ has been preferred over other coding systems because of 
its simplicity and in order to reutilize the same tool used in 
syndrome formation. In particular, independently of the TCQ 
system used to obtain sv{x), two TCQ systems have been 
tested, both based on 8-state trellises. 

1) Z/4Z-based TCQ. Given the target rate R, a regular 
TCQ based on the partition VL/AVL is designed (by 
dimensioning h) such that its normalized volume is 2^^ 
times less than the normalized volume relative to the 
TCQ system used in syndrome formation. 

2) Rate-distortion optimized TCQ. Once R is assigned, a 
finite-alphabet TCQ system achieving the rate [i?] (that 
uses 2^^^+^ reconstruction points [26]) is iteratively 
optimized for a set of 100 syndromes that serve as 
training sequences. The used algorithm is an adaptation 
to the trellis structure of the algorithm presented in [32]. 
In particular, all reconstruction levels are grouped into 
2 supersets [33] and the occurrences of any level into 
each superset at each iteration is used for codeword- 
length estimation. By varying the Lagrange multiplier 
A, the design can fit not only the rate R but any rate up 
to about [i?] . 

In both cases, a coding system was not actually designed, 
but the entropy H of the optimal path into the trellis, estimated 
using the 2 supersets as contexts, is assumed as achieved 
rate. On the other hand, it is reasonable that context-based 
variable-length coding based on the same contexts achieves 
exactly that rate. Fig. |9(a)| and Fig. |9(b)| show the average 
entropy (in the range BC-SNR e {9.0, 9.5, . . . , 19.0} dB, 
with 8-state syndrome formation) obtained in the two cases 
in correspondence of the target rates R G {0.5, 1.0, . . . , 3.0} 
bit/sample. Using Z/4Z-based TCQ the entropy is somewhat 
greater than desired, in particular at i? < 2 bit/sample, due 
to the impossibility to exactly cover the domain of sv{x) 
with polytopes that are translations of the same fundamental 
Voronoi region. Instead, with rate-distortion optimized TCQ, it 
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(a) Z/4Z-based TCQ 



3.5 



1.5 



0.5 1 1.5 2 2.5 3 3.5 

R (bit/sample) 

(b) rate-distortion optimized TCQ 

Fig. 9. Average entropy vs. target rate. In case of Z/4Z-based TCQ (a), the 
border effects cause the average entropy Hm to be actually greater than the 
desired target rate R; using rate-distoi1ion optimized TCQ (b) it is instead 
possible to exactly obtain any desired rate. 



is possible to achieve exactly the desired rate, paying the price 
of a slightly more complex encoder that requires non-uniform 
quantization and rate-distortion aware metric computation. 



C. Finite-Rate Volume Optimization 

In practice, by optimizing the value of A' > 1 it is not 
only possible to take into account for the non-optimality of 
the TCQ codes in the case of sv{x) being correctly received 
at the decoder, but also possible to take into account for the 
fact that at low rates q is not negligible w.rt. n, and hence 
Pe ^ 0. In fact, K is responsible for the normalized volume 
of the lattice A and, by increasing its value (at a fixed rate R) 

. the probabiHty of error = P[QA{n+q) ^ 0] ^ P[X- 

sv{X) j^X - {Sv{X) + Q)] reduces; 
• the granular error variance increases. 
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Fig. 10. Average error variance vs. volumetric factor, at BC-SNR equal 
to 14.0 dB (8-state, Z/4Z-based TCQ). The solid curves show the effect 
of varying the volumetric factor K on the final performance at two different 
target rates R. At low K, the total error is approximated by the overload error 
(whose equivalent variance is shown as the dashed curve); at high K, the 
total eiTor is approximated by the granular error (whose equivalent variance 
is shown as the dot-dashed curve). The circles indicate the optimum values 
of K, which decreases when R increases. 



When an error happens qt 
variance equals 



<loU 



and hence the total error 



a\^{\~P,)al+P,a\ 



(22) 



In Fig.[TOl experimental values of both terms of the right hand 
side of (|22] | are shown in function of K as the dot-dashed and 
the dashed curve respectively. It is clear that there exist an 
optimum value of the volumetric factor K, possibly different 
at each rate. The solid curve, i.e. the experimental value of 
the total error variance < cr^^ after MSB estimation (with 
in (I2TI ) hypothesized to equal ct^ ~ K<t^^2~'^^ - as shown 
later, the optimum value of K leads indeed to a negligible 
probability of error Pg), shows this effect. 

For each different code used in syndrome formation, the 
best value of K was found by a dichotomic-like search at each 
BC-SNR and at each target rate R. The results relative to the 
8-state system, reported in Fig. |ll(a)[ show that as expected 
K depends only on the target-rate, and is larger at lower rates, 
where there would be a significant amount of errors if K was 
too small. Correspondingly, the optimal value of P^ is constant 
at the various correlations, as shown in Fig. |ll(b)| 

The average values K,n and Pe„i of K and P^ over 
the various BC-SNR values are plotted against the average 
entropy achieved by the encoder in Fig. |12(a)| and Fig. |12(b)| 
respectively, using the three different TCQ codes for syndrome 
formation. Since by increasing the number of states the codes 
get closer to the capacity of the channel, it is expected, as 
actually obtained (see Fig. |12(a)| l, that the optimum values 
of K reduce; however, the same average probability of error 
seems to be actually required to achieve the best performance 
(see Fig. |12(b)| l. It is interesting to note that the optimal 
probability of error, which increases at low rates, is very close 
to the probability of error shown in [17] in correspondence of 
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Fig. 1 1 . Optimum volumetric factor (a) and corresponding error probability 
(b) vs. BC-SNR, at various experimental entropies (8-state, Z/4Z-based 
TCQ). 



the C-SNR that gives the optimal performance in the DISCUS 
system at both 1 and 2 bit/sample. 

D. Performance Analysis 

The average performance loss w.rt. the Wyner-Ziv bound 
Am is shown in function of the achieved transmission rate in 
Fig. [T3] and in Fig. [14] for Z/4Z-based syndrome coding and 
for rate-distortion optimized syndrome coding respectively. 
These values were obtained by averaging the experimental 
value of the performance loss A = (t^/((t,';2^^^'") over 
the various values of the BC-SNR in correspondence of the 
optimum value of K. Similarly, the error bars show the average 
95% confidence interval^ 

The experiments show that at any correlation and in the rate 
range R e [0.5, 3.0] bit/sample, the performance of the system 

'For each tuple (BC-SNR, R, K) the 95% confidence interval is estimated 
after up to 5000 independent simulations. 



— — 8-state 
— 64-state 
-e — 256-state 




(bit/sample) 
(b) 

Fig. 12. Average volumetric factor (a) and corresponding average probability 
of error (b) vs. experimental entropy, varying the number of states in syndrome 
formation (Z/4Z-based TCQ). While K decreases when the number of states 
increases, i.e. when the trellis codes are closer to capacity, the probability of 
error remains almost constant. 



is within 3 4 dB of the theoretical bound. In particular, at 
low rates, it is preferable to code the syndrome using rate- 
distortion optimized TCQ, which allows for a gain of more 
than 1 dB over the other method. 

To conclude the section, comparisons w.rt. to the DISCUS 
system are shown in Fig. |15(a)| and Fig. |15(b)| for the rates 
i? = 1 and i? = 2 bit/sample respectively. The immediate 
result is that by simply choosing the right scaling factor a 
the proposed system can adapt to any correlation and give 
the same performance loss, while the DISCUS system must 
be redesigned to adapt to different correlations. In addition, 
once the proposed system is designed for a certain additive 
BCH, no modifications are needed if the statistics of Y (and 
consequently the one of X) changes. Moreover, while only 
integer rates can be achieved by the DISCUS system, any rate 
can be achieved by distributed coding based on continuous- 
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Fig. 13. Average performance loss (w.r.t. the Wyner-Ziv bound) vs. experi- 
mental entropy, varying the number of states in syndrome formation (Z/4Z- 
based TCQ). 
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Fig. 14. Average performance loss (w.rt. the Wyner-Ziv bound) vs. exper- 
imental entropy, varying the number of states in syndrome formation (rate- 
distortion optimized TCQ). Rate-distortion optimized TCQ allows for better 
performance at low rates (note that in this case the expeiimental entropy 
equals the desired target rate). 

valued syndromes. This, again, is simply obtained by choosing 
the right value of b in case of Z/4Z-based syndrome coding 
or, with some increased complexity, by using an ad-hoc 
rate-distortion optimized TCQ, which finally leads to higher 
performance at low rates. 

While the complexity of the encoding operation is somewhat 
higher w.r.t. DISCUS, in which the discrete syndrome is com- 
puted on-the-fly after quantization, the decoding complexity is 
essentially the same, and hence it is perfectly feasible to use 
distributed coding based on continuous-valued syndromes in 
actual applications. 

VI. Conclusion and Future Work 

In this paper a coding system with side information at 
the decoder has been described that operates entirely in the 





- - Wyner-Ziv bound 




DISCUS (8-state) 




continuous-syndrome (8-state) 


\ ■ 


— ^ — continuous-syndrome (256-state) 











C-SNR (dB) 



(a) i? = 1 bit/sample 



-5 



-10 



-25 



-30 



-35 





- - Wyner-Ziv bound 




DISCUS (8-state) 




continuous-syndrome (8-state) 


N 


— ^ — continuous-syndrome (256-state) 


\ . 
\ 




\. 

\ 




\ 









'9 10 11 12 13 14 15 16 17 18 19 
C-SNR (dB) 



(b) R = 2 bit/sample 

Fig. 15. Performance comparison of the proposed algorithm (rate-distoi1ion 
optimized TCQ, ctJ = 1) vs. the DISCUS algorithm [17], at target rate 
-R = 1 (a) and R = 2 (b) bit/sample. The proposed algorithm outperforms 
the DISCUS system, which is optimal for a narrow range of coiTelations only. 

continuous domain. As a consequence of this approach, the 
system can be designed exactly according to the correlation 
between the source and the side information and according to 
the given transmission rate. Extensive experiments showed that 
the system achieves a good performance in case the correlation 
can be modeled by a backward Gaussian additive channel, 
even at low rates such as for example 0.5 bit/sample. 

Since in the proposed system the adaptation to the amount 
of correlation is simply done by means of a scaling factor, 
it is very interesting to investigate if on-the-fly adaptation 
to slightly time-varying correlation is possible, which would 
be of paramount importance in real applications. Moreover, 
since the actual syndrome coding operation is performed via 
traditional source coding, it is in principle possible to use 
scalable source coding tools to obtain a scalable distributed 
source coding algorithm, that could be applied in frameworks 
where the available transmission rate is not constant (such as in 
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wireless network-based applications). Both these observations 
represent a good starting point for future research. 
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